Image processing apparatus, image pickup apparatus and image processing program for correcting color blur

ABSTRACT

The image processing apparatus includes a determining part configured to determine, from a difference between information on color of a first pixel in a first image and information on color of a second pixel corresponding to the first pixel in a second image, whether or not the first image includes color blur due to defocus, the first and second images being generated by an image-pickup system and whose focus states are mutually different. The apparatus further includes a correcting part configured to perform on the first image a correction process that corrects the color blur determined by the determining part.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing technique thatcorrects color blur of an image generated by image pickup.

2. Description of the Related Art

Images generated by image pickup performed by image pickup apparatusessuch as digital cameras are often deteriorated by aberrations of animage taking optical system provided in the image pickup apparatus. Inparticular, longitudinal chromatic aberration of the image takingoptical system causes color blur in a color image.

Since lights of mutually different wavelengths have mutually differentrefractive indices, these lights entering an image taking optical systemform optical images at mutually different positions in an optical axisdirection of the image taking optical system. Image-pickup elements eachconverting an object image (optical image) formed by the image takingoptical system in the image pickup apparatus into electronic imageinformation generally have a planar light-receiving surface, and only anoptical image formed on the light-receiving surface is converted intosharp image information. Therefore, the longitudinal chromaticaberration generates, as the image information obtained by using theimage-pickup element, an image including color blur generated byoverlapping defocused color image components.

Japanese Patent Laid-Open No. 2008-85773 discloses a method forcorrecting color blur caused due to longitudinal chromatic aberration.This method obtains sharp (in-focus) images of respective wavelengthswhile changing a relative distance between an image taking opticalsystem and an image-pickup element such that lights of the respectivewavelengths form the sharp images on the image-pickup element, andcombines these sharp images to obtain a color image in which thelongitudinal chromatic aberration is corrected.

Moreover, so-called purple fringe also generates color blur, whichdeteriorates an image. The purple fringe is a phenomenon in which colorblur of purple is generated near a high luminance area or the like inthe image since point image distributions of respective wavelengths aremutually different due to longitudinal chromatic aberration, chromaticspherical aberration and chromatic comatic aberration. Japanese PatentLaid-Open No. 2006-115039 discloses a method for correcting such purplefringe. This method performs the correction by determining an area wherethe purple fringe is generated based on a distance from a luminancesaturated area and closeness to chroma and hue of a specific color, andby performing a predetermined spatial operation on the determined area.This method uses a fact that the purple fringe is easily generated nearthe luminance saturated area and easily becomes a specific color ofpurple.

The method disclosed in Japanese Patent Laid-Open No. 2008-85773combines the in-focus images of the respective wavelengths to correctthe longitudinal chromatic aberration, which enables improvement ofimage quality. Moreover, the method disclosed in Japanese PatentLaid-Open No. 2006-115039 supposes the area where the purple fringe isgenerated and corrects the purple fringe to some degree on the basis ofthe supposition.

However, there are actually some cases where the methods disclosed inJapanese Patent Laid-Open Nos. 2008-85773 and 2006-115039 cannot correctthe color blur well. For example, the methods cannot correct well colorblur generated in an image obtained by image pickup for athree-dimensional object. Image pickup apparatuses generally focus onone plane (in-focus point) in an object space by an autofocus functionor a manual focus operation and then perform image pickup. In such imagepickup, in a case where an object including the plane is athree-dimensional object, object distances at respective angles of vieware mutually different. In this case, a relatively sharp image of thein-focus point is obtained, but defocused images according to the objectdistances are obtained at out-of-focus points. Since the longitudinalchromatic aberration, the chromatic spherical aberration and thechromatic comatic aberration are varied according to the objectdistance, the color blur generated in the defocused image is variedaccording to a distance from the in-focus point. Such color blur beingvaried according to the object distance is hereinafter referred to as“color blur due to defocus”. The method disclosed in Japanese PatentLaid-Open No. 2008-85773 can correct the color blur due to thelongitudinal chromatic aberration for only an in-focus image of anobject located at a specific distance, but cannot correct the color blurdue to defocus. The method disclosed in Japanese Patent Laid-Open No.2006-115039 cannot provide a sufficient effect to correct the color blurdue to defocus since the method utilizes a tendency of generation of thecolor blur at the in-focus point.

SUMMARY OF THE INVENTION

The present invention provides an image processing apparatus, an imagepickup apparatus and an image processing program which are capable ofcorrecting the color blur due to defocus well.

The present invention provides as one aspect thereof an image processingapparatus including a determining part configured to determine, from adifference between information on color of a first pixel in a firstimage and information on color of a second pixel corresponding to thefirst pixel in a second image, whether or not the first image includescolor blur due to defocus, the first and second images being generatedby an image-pickup system and whose focus states are mutually different,and a correcting part configured to perform on the first image acorrection process that corrects the color blur determined by thedetermining part.

The present invention provides as another aspect thereof an image pickupapparatus including an image-pickup system configured to perform imagepickup to generate images, and an image processing apparatus. The imageprocessing apparatus includes a determining part configured todetermine, from a difference between information on color of a firstpixel in a first image and information on color of a second pixelcorresponding to the first pixel in a second image, whether or not thefirst image includes color blur due to defocus, the first and secondimages being generated by the image-pickup system and whose focus statesare mutually different, and a correcting part configured to perform onthe first image a correction process that corrects the color blurdetermined by the determining part.

The present invention provides as still another aspect thereof an imageprocessing program causing a computer to perform image processing. Theprogram including a step of obtaining a first image and a second imagewhich are generated by an image-pickup system and whose focus states aremutually different, and a step of determining, from a difference betweeninformation on color of a first pixel in a first image and informationon color of a second pixel corresponding to the first pixel in a secondimage, whether or not the first image includes color blur due todefocus, and a step of performing on the first image a correctionprocess that corrects the color blur determined by the determining step.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of an image pickupapparatus that is Embodiment 1 of the present invention.

FIG. 2 is a flowchart showing an image-pickup process in Embodiment 1.

FIG. 3 shows a situation of image pickup in Embodiment 1.

FIG. 4 shows a captured image in Embodiment 1.

FIG. 5 shows color information in the captured image in Embodiment 1.

FIG. 6 shows chromatic aberration in Embodiment 1.

FIG. 7 shows a color space in Embodiment 1.

FIG. 8 shows determination of a color blur area in Embodiment 1.

FIG. 9 shows determination of a color blur amount in Embodiment 1.

FIG. 10 shows a configuration of an image processing apparatus that isEmbodiment 3 of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Exemplary embodiments of the present invention will hereinafter bedescribed with reference to the accompanying drawings.

Embodiment 1

FIG. 1 shows a configuration of an image pickup apparatus that isEmbodiment 1 of the present invention. An image taking optical system101 includes an aperture stop 101 a that changes its aperture diameterto adjust a light amount (F number) and a focus lens 101 b that moves inan optical axis direction for focusing. A position of the focus lens 101b is controlled by an autofocus (AF) function and a manual focusoperation. The image taking optical system 101 causes light from anobject (not shown) to form an object image that is an optical image. Theobject image is converted into an electric signal by an image-pickupelement 102 constituted by a photoelectric conversion element such as aCCD sensor and a CMSO sensor.

The electric signal (analogue signal) output from the image-pickupelement 102 is converted into a digital signal by an A/D converter 103,and the digital signal is input to an image processor 104. The imageprocessor 104 performs various signal processing on the input digitalsignal to generate image data. The image data is hereinafter referred toas a “captured image”. The image taking optical system 101, theimage-pickup element 102 and the image processor 104 constitute animage-pickup system.

The image processor (image processing apparatus) 104 performs acorrection process of the color blur due to defocus on the capturedimage. The correction process of the color blur due to defocus will bedescribed later, and is hereinafter simply referred to as “color blurcorrection”. Specifically, the image processor 104 acquires informationon a state of the image taking optical system 101, which is hereinafterreferred to as an “image-pickup condition”, from a state detecting part107. Then, the image processor 104 selects color variation informationcorresponding to the image-pickup condition from a memory 108 to performthe color blur correction on the input captured image. The imageprocessor 104 corresponds to a correcting part.

Next, description will be made of an image-pickup process including thecolor blur correction in the image pickup apparatus of this embodimentwith reference to a flowchart shown in FIG. 2. A system controller 110shown in FIG. 1 and constituted by a microcomputer performs theimage-pickup process according to computer programs. The image takingoptical system 101 is a zoom lens whose focal length is variable.

When an image-pickup start signal is input in response to a user'soperation, the system controller 110 moves the focus lens 101 b in theimage taking optical system 101 to an in-focus position for a mainobject to be captured at a current zoom position (step S1). This focuscontrol is performed through an image taking optical system controller106 shown in FIG. 1. Thereby, an optical image of the main object isclearly formed on a light-receiving surface of the image-pickup element102. The optical image of the main object is hereinafter referred to asa “main object image”, and the in-focus position of the focus lens 101 bfor the main object is hereinafter referred to as a “first focusposition”.

Next, the system controller 110 acquires information on the image-pickupcondition from the state detecting part 107 (step S2). The image-pickupcondition includes a zoom position (focal length), an aperture value (Fnumber) and a position of the focus lens 101 b, which show the state ofthe image taking optical system 101.

Next, the system controller 110 calculates an object distance rangewhere the color blur due to defocus may be generated by using theacquired information on the image-pickup condition. Then, the systemcontroller 110 decides a second focus position that is a position of thefocus lens 101 b different from the first focus position in thecalculated object distance range (step S3).

Next, the system controller 110 performs first image pickup in a statewhere the focus lens 101 b is located at the first focus position, andcauses the image processor 104 to generate a first captured image (firstimage) that is a captured image before the color blur correction (stepS4). This first captured image is defined as a reference image in thecolor blur correction in this embodiment. The system controller 110causes the image processor 104 to temporarily store the reference imagetherein.

When the first image pickup has been thus completed, the systemcontroller 110 moves the focus lens 101 b to the second focus positionthrough the image taking optical system controller 106 (step S5).

Then, the system controller 110 performs second image pickup in a statewhere the focus lens 101 b is located at the second focus position, andcauses the image processor 104 to generate a second captured image(second image) that is also an image before the color blur correctionbut its focus state is different from that of the first captured mage(step S6). This second captured image is defined as a comparison imagein the color blur correction in this embodiment. The system controller110 causes the image processor 104 to temporarily store the comparisonimage therein.

Next, the system controller 110 performs the color blur correctionaccording to an image processing program that is part of theabove-described computer programs. The image processing program may be acomputer program separate from the above-described computer programs.

The system controller 110 detects information on color for correspondingpixels (first pixels and second pixels corresponding to each other) inthe reference and comparison images stored in the image processor 104(step S7). The information on color in this embodiment includes chromaand hue. However, the information on color may include at least one ofcolor difference, chroma and hue. The information on color ishereinafter simply referred to as “color information”.

In order to detect the chroma and the hue, the system controller 110converts the reference image and the comparison image which arerespectively constituted by R, G and B components into data in a colorspace that has a luminance component and a color component separately.Although the color space of the converted data may be, for example,YCbCr, CIE L*a*b* or the like, this embodiment uses the CIE L*a*b* colorspace.

The chroma C and the hue θ that are elements of the color information inthe CIE L*a*b* color space are decided by chromaticity a*, b*. The colorinformation at a pixel of coordinates (x, y) in the reference image isdenoted by chroma C1(x, y) and hue θ1(x, y). Moreover, the colorinformation at a pixel of coordinates (x, y) in the comparison image isdenoted by chroma C2(x, y) and hue θ2(x, y). The pixel (first pixel) ofthe coordinates (x, y) in the reference image and the pixel (secondpixel) of the coordinates (x, y) in the comparison image are“corresponding pixels” whose coordinates are identical to each other.

Next, the system controller 110 as a determining part performs colorblur determination (or color blur area determination) that is a firstprocess in the color blur correction (step S8).

Specifically, the system controller 110 evaluates a variation amount ofthe color information at the corresponding pixels (that is, a differencebetween the color information at the first pixel and the colorinformation at the second pixel) in the reference image and thecomparison image whose focus states are mutually different because thepositions of the focus lens 101 b in the first image pickup and thesecond image pickup are mutually different, to determine whether or notthe color blur due to defocus is generated. This makes it possible todetermine an area where the color blur is generated. The area where thecolor blur is generated is hereinafter referred to as a “color blurarea”. The determination of whether or not the color blur is generatedand the determination of the color blur area can be made by using thefollowing expressions. Jc and Jθ respectively represent variationamounts of the color information, that is, the chroma and the hue, thechroma variation amount Jc and the hue variation amount Jθ being definedby the following expressions:Jc=C1(x,y)−C2(x,y)Jθ=θ1(x,y)−θ2(x,y).

When both the chroma variation amount Jc and the hue variation amount Jθare 0, the variation (difference) of the color information due to thedifference of the focus state is not generated, which means the colorblur due to defocus is not generated. On the other hand, when at leastone of the chroma variation amount Jc and the hue variation amount Jθ isother than 0, the color blur due to defocus is generated. Thus, thedetermination of generation of the color blur and the determination ofthe color blur area can be made.

Description will be made of examples of a case where the chroma and huevariation amounts are 0 and a case where the chroma and hue variationamounts are other than 0 with reference to FIGS. 3 to 8. FIG. 3 shows astate where an image pickup apparatus 601 performs image pickup of blacksticks 611 to 615 that are objects placed obliquely from a far pointtoward a near point. The image pickup apparatus 601 is focused on theobject 613. FIG. 4 shows a captured image captured in this state.Moreover, FIG. 5 is a cross-sectional view showing the color informationin areas (shown by arrows) respectively including the objects 613 and614 shown in FIG. 4.

The object 613 in FIG. 5 is located at an object distance (in-focusdistance) on which the image pickup apparatus 601 is focused (in-focus),and the object 614 is located farther than the in-focus distance. Anupper part in FIG. 5 shows the color information in the areas includingthe objects 613 and 614 in the reference image. A middle part thereinshows the color information in the areas including the objects 613 and614 in the comparison image captured in a state where an in-focusdistance is set to a distance between the object 613 and the object 612.

In the reference image, a G component is in focus for the object 613located at the in-focus distance. Therefore, a defocus (out-of-focus)amount at an edge of the G component is smaller than those of an Rcomponent and a B component. The R component and the B component arefurther defocused by mutually nearly equal defocus amounts than the Gcomponent. An image formation state corresponding to the above statewill be described with reference to FIG. 6.

FIG. 6 shows R, G and B rays whose wavelengths are mutually differentand which enter the image-pickup element 102 from objects located at thein-focus distance (shown at a middle part), at a near distance nearerthan the in-focus distance (shown is an upper part) and at a fardistance farther than the in-focus distances (shown at a lower part)through an image taking optical system. At the in-focus distance, the Gray forms a clear (sharp) object image of the G component on theimage-pickup element 102, and however the R ray and the B ray havedivergence on the image-pickup element 102, that is, object images ofthe R and G components are defocused.

On the other hand, at the far distance, the R and B rays form clearobject images of the R and B components on the image-pickup element 102,but the G ray diverges on the image-pickup element 102 and therefore anobject image of the G element is defocused. Furthermore, at the neardistance, each of the R, G and B rays diverges, and in particular the Rand B rays diverge further than the G ray.

FIG. 6 shows longitudinal chromatic aberration of the image takingoptical system of the image pickup apparatus 601, and a similarphenomenon is generated in the variation of the color information atpositions away from an optical axis of the image taking optical systemin an image-pickup angle of view though an asymmetric property isgenerated in a basic image formation characteristic of the image takingoptical system.

In the comparison image shown at the middle part in FIG. 5, animage-forming state of the object 613 is closer to an image-formingstate when the object 613 is located at the far distance than that whenthe object 613 is located at the in-focus distance (as shown in thereference image), and the defocus amount of the object image of the Gcomponent is mainly increased. Moreover, in the comparison image, animage-forming state of the object 614 becomes an image-forming statewhen the object 614 is located at a farther distance, and the defocusamounts of the object images of the R and G components are mainlyincreased.

Description will be made of the color information at coordinates 1(x1,y1) of a pixel in the vicinity of the object 614 in FIG. 5, thecolor information at coordinates 2 (x2,y2) of a pixel in the vicinity ofa center of the object 614 and the color information at coordinates 3(x3,y3) of a pixel in the vicinity of the object 613. FIG. 7 shows, asan example, the color information in a*b* space at the coordinates 1(x1,y1) in the reference image and the color information in thecomparison image. As clear from FIG. 5, at the coordinates 1 (x1,y1), adifference of the defocus amounts in the reference and comparison imagesis caused due to variations of the focus states of the R, G and Bcomponents. Therefore, as shown in FIG. 7, the chroma and the hue areboth varied between in the reference image and in the comparison image.This variation corresponds to a defocus characteristic of the imagetaking optical system 101, and a similar variation thereto is inevitablycaused between two captured images whose focus states for an objectlocated at a specific distance are mutually different. Similarly, thechroma and the hue are also varied at the coordinates 3 (x3,y3). On theother hand, the chroma and the hue are not varied at the coordinates 2(x2,y2) since a difference of the defocus amounts due to the variationsof the focus states of the R, G and B components is not caused as clearfrom FIG. 5.

The chroma and hue variation amounts at each of the coordinates areobtained by calculating the difference of the color information at thecorresponding pixels in the reference image and the comparison image byusing the following expressions:Jc(x1,y1)=C1(x1,y1)−C2(x1,y1)Jθ(x1,y1)=θ1(x1,y1)−θ2(x1,y1)Jc(x2,y2)=C1(x2,y2)−C2(x2,y2)Jθ(x2,y2)=θ1(x2,y2)−θ2(x2,y2).

As clear from FIGS. 5 and 7, the chroma and hue variation amounts are asfollows:Jc(x1,y1)>0Jθ(x1,y1)>0Jc(x2,y2)=0Jθ(x2,y2)=0.

In other words, the case where the chroma and hue variation amounts Jcand Jθ at certain coordinates are other than 0 shows that the color blurdue to defocus is generated at the certain coordinates, and the casewhere both the chroma and hue variation amounts Jc and Jθ at certaincoordinates are 0 shows that no color blur due to defocus is generatedat the certain coordinates. Performing such determination on the entirereference image enables determination of the color blur area where thecolor blur due to defocus is generated in the reference image as shownin FIG. 8.

When the color blur area exists, the system controller 110 performscolor blur amount estimation as a second process in the color blurcorrection (step S9). When no color blur area exists in the referenceimage, the system controller 110 determines that no color blur due todefocus is generated, and then ends the color blur correction.

Next, description will be made of the color blur amount estimation. Asdescribed above, the variation of the color information caused by thecolor blur due to defocus between the reference and comparison imageswhose focus states are mutually different corresponds to the defocuscharacteristic of the image taking optical system 101. In other words,the variation of the color information corresponds to a fact that, undera certain image-pickup condition, the longitudinal chromatic aberration,the chromatic comatic aberration and the chromatic spherical aberrationare varied according to a shift amount (hereinafter referred to as a“defocus amount (defocus distance)”) d of the object distance from anobject distance position on which the image taking optical system isfocused. The object distance position on which the image taking opticalsystem is focused is hereinafter referred to as the “in-focus referenceposition”.

FIG. 9 shows a relationship between the defocus amount d and the hue θ,and a relationship between the defocus amount d and the chroma C whendefining the first focus position for the image taking optical system101 of this embodiment as the in-focus reference position. As understoodfrom these relationships, the variation of the color information in thecolor blur area where the color blur due to defocus is generated can beexpressed by a function whose variable is the defocus amount d.

This embodiment defines, as the functions of the chroma C and the hue θwhose variable is the defocus amount d, a chroma variation functionfc(d) and a hue variation function fθ(d). Specifying these two functionsmakes it possible to correct the color blur in the reference imageeasily.

Specifying the chroma variation function fc(d) and the hue variationfunction fθ(d) requires in principal information on a distance to anobject (object distance information), information on a color of theobject (object color information) and information on variation of anoptical performance of the image taking optical system 101 associatedwith variation of a focus state according to an image-pickup condition(optical performance variation information). Although the objectdistance information can be obtained by various measuring methods, mostof them are not a measuring method easy to be implemented because ofrequirement of a special device other than the image taking opticalsystem or the like.

Thus, this embodiment obtains information equivalent to the objectdistance information and the object color information, from thevariation amount of the color information between the reference andcomparison images whose focus states are made to be mutually differentby a known defocus amount.

Specifically, the system controller 110 acquires the optical performancevariation information that is information relating to the variation ofthe optical performance of the image taking optical system 101associated with the variation of the focus state under the image-pickupcondition at a time of acquisition (generation) of the reference imagefrom the memory 108 in which the information is stored in advance. Theoptical performance variation information shows variations of chromaticaberrations such as longitudinal chromatic aberration, chromatic comaticaberration and chromatic spherical aberration. Variation rate of thechroma variation function fc(d) and the hue variation function fθ(d)with respect to the defocus amount d can be obtained from the opticalperformance variation information.

Then, the system controller 110 performs calculation for estimating thechroma variation function fc(d) and the hue variation function fθ(d) byusing as parameters the optical performance variation information, thecolor information of each pixel in the reference and comparison images,C1, C2, θ1, θ2, the object distance information and the object colorinformation. That is, the system controller 110 estimates the chromavariation function fc(d) and the hue variation function fθ(d) such thatgraphs thereof shown in FIG. 9 pass C1, C2, θ1 and θ2.

Although this embodiment describes the case where the chroma variationfunction fc(d) and the hue variation function fθ(d) are estimated byusing two images that are the reference and comparison images, increaseof the number of the comparison images can improve estimation accuracy.

Finally, the system controller 110 calculates an estimated object colorand an estimated color blur amount from the chroma variation functionfc(d) and the hue variation function fθ(d), which corresponds to a thirdprocess in the color blur correction. Then, the system controller 110causes the image processor 104 to perform a color correction process forreducing the color blur due to defocus on the reference image by usingthe estimated object color and the estimated color blur amount (stepS10).

A lower part of FIG. 5 shows images after the color blur correction.Although the color blur correction is made with reference to the Gcomponent in FIG. 5, it may be made with reference to the R component orthe B component.

Moreover, the above-described processes may be performed on pixel bypixel basis or may be performed on each area including plural pixelsaccording to a purpose of speed-up or the like.

After completion of the color blur correction performed on each pixel inthe reference image, the system controller 110 displays an output imageafter the color blur correction on a displaying part 105 shown in FIG. 1or records it to a recording medium 109 such as a semiconductor memory,and then ends the image-pickup process.

Although this embodiment described the case where the image pickupapparatus is integrally provided with the image taking optical system101, the image pickup apparatus may be constituted by an interchangeablelens including an image taking optical system and an image pickupapparatus main body to which the interchangeable lens is detachablyattached.

Embodiment 2

Next, description will be made of a second embodiment (Embodiment 2) ofthe present invention. This embodiment will describe an image pickupapparatus that generates a captured image by using as an image-pickupelement 102 a Bayer-type image-pickup element in which R, G and B colorfilters are arranged in Bayer arrangement.

A signal obtained from each pixel of the Bayer-type image-pickup elementhas a signal value of one of R, G and B components. Image data producedby using such signals is referred to as “RAW image data”. Each pixel ofthe RAW image data has only a signal value of one color component, sothat this embodiment performs a color synthesis process (demosaicingprocess) to provide an RGB value to each pixel of an output image. Thecolor synthesis process performs an interpolation process using signalvalues of plural pixels (peripheral pixels) around a target pixel toprovide the RGB value to the target pixel.

However, since the RAW image data has only one signal value at eachpixel, color information cannot be extracted therefrom. Therefore, thisembodiment will describe color blur correction for a case where both thereference image and the comparison image are RAW image data. Aconfiguration of the image pickup apparatus of this embodiment isbasically the same as that of the image pickup apparatus of Embodiment 1(shown in FIG. 1), and therefore components in this embodiment common tothose of the image pickup apparatus of Embodiment 1 are denoted by thesame reference numerals as those in Embodiment 1.

First of all, the system controller 110 extracts a feature amountrelating to color of a target pixel (color information acquisitiontarget pixel) by using the signal values of the peripheral pixels aroundthe target pixel in order to acquire color information of each pixel.The extraction of the feature amount may be made by the above-describedcolor synthesis process or a simplified color synthesis process.Moreover, the chroma and the hue which has been described in Embodiment1 may be used as the color information. Furthermore, any one of colorspaces being expressed by YCbCr, XYZ, Yuv and JCh may be selectivelyused.

Thus, the feature amount relating to color can be expressed by variousexpression methods, and the color information of the pixel can beexpressed by an arbitrary expression method. The “color information” inthis embodiment may be expressed by any expression methods as long as itrepresents the feature amount relating to color. This also applies tothe case where one pixel originally has the R, G and B values asabove-described Embodiment 1.

The system controller 110 once stores the color information of eachpixel thus acquired. Then, the system controller 110 determines, as inEmbodiment 1, the variation of the color information of thecorresponding pixels in the reference and comparison images whose focusstates are mutually different to determine the color blur area.

The determination of the color blur area may be made by using acomparison method corresponding to the color space such as YCbCr.Additionally, it may be made by using uv of Yuv or uv of Luv.

Next, the system controller 110 estimates the color blur amount in thecolor blur area by using a similar method to that used in Embodiment 1to correct the color blur.

The color blur correction can be made by other methods such as a spatialoperation with interpolation using pixels adjacent to the target pixel.An amount of the color blur correction may be decided according to arequired image quality level of the output image, a permissible loadvalue of the process or the like.

In addition, in order to more accurately perform the color blurcorrection described in Embodiments 1 and 2, matching of coordinates ofthe reference and comparison images may be made with reference toextracted edge portions of these images.

Embodiment 3

Although Embodiments 1 and 2 described the case where the imageprocessor 104 as the image processing apparatus is provided in the imagepickup apparatus, an alternative embodiment of the present invention isnot limited thereto.

For example, as shown in FIG. 10, image date generated by image pickupperformed by an image pickup apparatus 1401 is transmitted to a personalcomputer 1402. A method for transmitting the image to personal computer1402 may be any of a cable method and a wireless method through Internetor LAN.

Then, the personal computer 1402 performs the color blur correction atsteps S7 to S10 in the flowchart of FIG. 2. In this case, the personalcomputer 1402 serves as an image processing apparatus of an embodimentof the present invention.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2010-021015, filed on Feb. 2, 2010, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: adetermining part configured to determine, from a difference betweeninformation on color of a first pixel in a first image and informationon color of a second pixel corresponding to the first pixel in a secondimage, whether or not the first image includes color blur due todefocus, wherein the color blur due to defocus is color blur beinggenerated by variation of longitudinal chromatic aberration, chromaticspherical aberration and chromatic comatic aberration of an image-pickupsystem depending on object distance, the first and second images beinggenerated by image pickup of a three-dimensional object through theimage-pickup system set in mutually different focus states; and acorrecting part configured to perform on the first image a correctionprocess that corrects the color blur determined by the determining part,wherein the correcting part is configured to calculate a variationfunction including as a parameter a defocus amount by using informationon the variation of at least one of the longitudinal chromaticaberration, chromatic spherical aberration and chromatic comaticaberration corresponding to variation of the focus state of theimage-pickup system and the information on color of each pixel in thefirst and second images; estimate an amount of the color blur by usingthe variation function; and perform the correction process based on theestimated amount of the color blur.
 2. An image processing apparatusaccording to claim 1, wherein the information on color includesinformation on chroma and hue.
 3. An image processing apparatusaccording to claim 1, wherein each of the first image and the secondimage is of a substantive in-focus state.
 4. An image processingapparatus according to claim 1, wherein the color blur due to defocus isgenerated by variation of a difference between blur of a green colorcomponent of the first image and blur of one of a red color componentand a blue color component of the first image, depending on a defocusamount.
 5. An image processing apparatus according to claim 1, whereinthe color blur due to defocus is generated by overlapping of defocusedcolor image components.
 6. An image pickup apparatus comprising: animage-pickup system configured to perform image pickup to generateimages; and an image processing apparatus, wherein the image processingapparatus comprising: a determining part configured to determine, from adifference between information on color of a first pixel in a firstimage and information on color of a second pixel corresponding to thefirst pixel in a second image, whether or not the first image includescolor blur due to defocus, wherein the color blur due to defocus iscolor blur being generated by variation of longitudinal chromaticaberration, chromatic spherical aberration and chromatic comaticaberration of the image-pickup system depending on object distance, thefirst and second images being generated by image pickup of athree-dimensional object through the image-pickup system set in mutuallydifferent focus states; and a correcting part configured to perform onthe first image a correction process that corrects the color blurdetermined by the determining part, wherein the correcting part isconfigured to calculate a variation function including as a parameter adefocus amount by using information on the variation of at least one ofthe longitudinal chromatic aberration, chromatic spherical aberrationand chromatic comatic aberration corresponding to variation of the focusstate of the image-pickup system and the information on color of eachpixel in the first and second images; estimate an amount of the colorblur by using the variation function; and perform the correction processbased on the estimated amount of the color blur.
 7. An image processingmethod comprising: obtaining a first image and a second image which aregenerated by image pickup of a three-dimensional object through animage-pickup system set in mutually different focus states; determining,from a difference between information on color of a first pixel in thefirst image and information on color of a second pixel corresponding tothe first pixel in the second image, whether or not the first imageincludes color blur due to defocus, wherein the color blur due todefocus is color blur being generated by variation of longitudinalchromatic aberration, chromatic spherical aberration and chromaticcomatic aberration of the image-pickup system depending on objectdistance; performing on the first image a correction process thatcorrects the determined color blur; calculating a variation functionincluding as a parameter a defocus amount by using information on thevariation of at least one of the longitudinal chromatic aberration,chromatic spherical aberration and chromatic comatic aberrationcorresponding to variation of the focus state of the image-pickup systemand the information on color of each pixel in the first and secondimages; estimating an amount of the color blur by using the variationfunction; and performing the correction process on a basis of theestimated amount of the color blur.
 8. A non-transitorycomputer-readable storage medium storing a program that, when executedby a computer, causes the computer to perform a method, the methodcomprising: obtaining a first image and a second image which aregenerated by image pickup of a three-dimensional object through animage-pickup system set in mutually different focus states; determining,from a difference between information on color of a first pixel in thefirst image and information on color of a second pixel corresponding tothe first pixel in the second image, whether or not the first imageincludes color blur due to defocus, wherein the color blur due todefocus is color blur being generated by variation of longitudinalchromatic aberration, chromatic spherical aberration and chromaticcomatic aberration of the image-pickup system depending on objectdistance; performing on the first image a correction process thatcorrects the determined color blur; calculating a variation functionincluding as a parameter a defocus amount by using information on thevariation of at least one of the longitudinal chromatic aberration,chromatic spherical aberration and chromatic comatic aberrationcorresponding to variation of the focus state of the image-pickup systemand the information on color of each pixel in the first and secondimages; estimating an amount of the color blur by using the variationfunction; and performing the correction process on a basis of theestimated amount of the color blur.